Least Squares Support Vector Machines: an Overview

Support Vector Machines is a powerful methodology for solving problems in nonlinear classification, function estimation and density estimation which has also led recently to many new developments in kernel based learning in general. In these methods one solves convex optimization problems, typically quadratic programs. We focus on Least Squares Support Vector Machines which are reformulations to standard SVMs that lead to solving linear KKT systems. Least squares support vector machines are closely related to regularization networks and Gaussian processes but additionally emphasize and exploit primaldual interpretations from optimization theory. In view of interior point algorithms such LS-SVM KKT systems can be considered as a core problem. Where needed the obtained solutions can be robustified and/or sparsified. As an alternative to a top-down choice of the cost function, methods from robust statistics are employed in a bottom-up fashion for further improving the estimates. We explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. The framework is further extended towards unsupervised learning by considering PCA analysis and its kernel version as a one-class modelling problem. This leads to new primal-dual support vector machine formulations for kernel PCA and kernel canonical correlation analysis. Furthermore, LS-SVM formulations are mentioned towards recurrent networks and control, thereby extending the methods from static to dynamic problems. In general, support vector machines may pose heavy computational challenges for large data sets. For this purpose, we propose a method of Fixed Size LS-SVM where the estimation is done in the primal space in relation to a Nyström sampling with active selection of support vectors and we discuss extensions to committee networks. The methods will be illustrated by several benchmark and real-life applications.